-
Notifications
You must be signed in to change notification settings - Fork 422
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for channel tabs and channel tags #1082
Add support for channel tabs and channel tags #1082
Conversation
200fcc1
to
53df9d6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Partial review; I had some time to look at the first few commits (up until the YouTube one, where I stopped at YoutubeChannelExtractor
)
extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelHelper.java
Show resolved
Hide resolved
extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelHelper.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a few small things to change, but overall everything is fine. If you don't have objections, I can fix everything myself and then merge. Thank you again to you and Theta-Dev for the effort on this PR!
Btw, for the future try to keep the commit subject under 50 characters: https://stackoverflow.com/q/2290016
...n/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeChannelExtractor.java
Show resolved
Hide resolved
...ava/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeChannelTabExtractor.java
Show resolved
Hide resolved
...ava/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeChannelTabExtractor.java
Show resolved
Hide resolved
...schabi/newpipe/extractor/services/youtube/extractors/YoutubeChannelTabPlaylistExtractor.java
Show resolved
Hide resolved
extractor/src/test/java/org/schabi/newpipe/extractor/ExtractorAsserts.java
Show resolved
Hide resolved
...est/java/org/schabi/newpipe/extractor/services/peertube/PeertubeChannelTabExtractorTest.java
Show resolved
Hide resolved
...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java
Show resolved
Hide resolved
* </pre> | ||
* </p> | ||
*/ | ||
public class ReadyChannelTabListLinkHandler extends ListLinkHandler { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did we decide on a better name or not in the end?
This commit introduces the following breaking changes: - Three new classes have been added: - ChannelTabExtractor, class extending ListExtractor<InfoItem>, which extracts InfoItems from a channel tab; - ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from a ChannelTabExtractor and returns them as a ChannelTabInfo; - ChannelTabs, an immutable class containing all supported channel tabs. - StreamingService implementations must implement new methods returning a channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a ChannelTabExtractor (getChannelTabExtractor); - ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem> and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>; - ChannelExtractor and ChannelInfo have now getters and/or setters of tabs. Co-authored-by: ThetaDev <[email protected]> Co-authored-by: Stypox <[email protected]>
Tags' getters and/or setters have been added in ChannelExtractor and ChannelInfo to do so. Co-authored-by: ThetaDev <[email protected]>
…y getId and getUrl methods of LinkHandlerFactory and its base implementations This change advertise to clients that channel tabs' link handler factories can return an UnsupportedOperationException when a tab provided to them is unsupported.
This class makes easier for LinkHandlerFactory implementations to declare an UnsupportedOperationException.
…fo.getTabs() when a specific tab's data has already been fetched This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving clients data, energy and time by helping to reduce duplicate requests. Co-authored-by: Stypox <[email protected]>
Support of channels and videos has been added for accounts and support of videos and playlists has been added for video channels. The following changes have been also done: - collectStreamsFrom method in PeertubeParsingHelper has been renamed to collectItemsFrom; - PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to ChannelExtractor's new inheritance; - the declaration of the UnsupportedOperationException exception thrown has been added to the service's LinkHandlers; - a channel tab LinkHandlerFactory has been added, PeertubeChannelTabLinkHandlerFactory; - all service's LinkHandlers are now using properly the singleton pattern. Co-authored-by: ThetaDev <[email protected]> Co-authored-by: Stypox <[email protected]>
Support of tracks and albums has been added for artists. Also use the singleton pattern and add the declaration of the UnsupportedOperationException exception to the service's LinkHandlers and improved some code in the files changed. Co-authored-by: ThetaDev <[email protected]> Co-authored-by: Stypox <[email protected]>
Support of tracks, playlists and albums has been added for users. Also add the declaration of the UnsupportedOperationException exception to the service's LinkHandlers. Co-authored-by: ThetaDev <[email protected]> Co-authored-by: Stypox <[email protected]>
…atterns This is required to parse duration of YouTube's reelItemRenderers, returned only inside accessibility data. Co-authored-by: ThetaDev <[email protected]>
…annels Support of tags and videos, shorts, live, playlists and channels tabs has been added for non-age restricted channels. Age-restricted channels are now also supported and always returned the videos, shorts and live tabs, accessible using system playlists. These tabs are the only ones which can be accessed using YouTube's desktop website without being logged-in. The videos channel tab parameter has been updated to the one used by the desktop website and when a channel extraction is fetched, this tab is returned in the list of tabs as a cached one in the corresponding link handler. Visitor data support per request has been added, as a valid visitor data is required to fetch continuations with contents on the shorts tab. It is only used in this case to enhance privacy. A dedicated shorts UI elements (reelItemRenderers) extractor has been added, YoutubeReelInfoItemExtractor. These elements do not provide the exact view count, any uploader info (name, URL, avatar, verified status) and the upload date. All service's LinkHandlers are now using the singleton pattern and some code has been also improved on the files changed. Co-authored-by: ThetaDev <[email protected]> Co-authored-by: Stypox <[email protected]>
…s with the Test JUnit annotation These changes should help to detect tests as tests, when running a subset of tests or all tests. They should be also implemented in these interfaces' implementations (new and existing ones). Co-authored-by: ThetaDev <[email protected]>
…returned by a channel extractor Only the first content filter of the ListLinkHandler instances provided is used when collecting all channel tabs of the ListLinkHandler list, as channel tabs implementations only use one content filter per ListLinkHandler instance. Co-authored-by: ThetaDev <[email protected]>
… on tests Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest have been also fixed. Co-authored-by: ThetaDev <[email protected]>
… on tests Co-authored-by: ThetaDev <[email protected]>
… on tests Co-authored-by: ThetaDev <[email protected]>
…es on tests Co-authored-by: ThetaDev <[email protected]>
…d channels tabs and tags changes on tests Co-authored-by: ThetaDev <[email protected]>
53df9d6
to
e7d6409
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aha! I took back ownership of commits! I rebased and tested locally, everything works fine. I will open a PR for the small suggestions I proposed above, as they are not blocking.
Already done by someone else: Add support for channel tabs NewPipe#9182, which needs to be updated to the latest changes made in this pull request (see below).
This PR supersedes #951 by reusing the changes from @Theta-Dev and @Stypox and applying the latest requested changes where relevant. Thanks a lot for your original work!
It introduces support of channel tabs and tags. The channel tab changes are breaking changes, as
ChannelExtractor
is now only anExtractor
abstract implementation instead of aListExtractor
one: this means thatChannelExtractor
do not provideStreamInfoItem
anymore. They can be extracted from channel tabs instead.Channel tabs (which are strings to be compliant with the current
LinkHandler
filtering and sorting system) need to be fetched separately from channels. A dynamic list of available tabs is returned byChannelInfo
andChannelExtractor
with the methodgetTabs
. The tab list is a list ofListLinkHandler
s, which can be used to fetch tabs using thegetChannelTabExtractor
method.In some cases, tab data is sent with the channel data, so fetching tabs would require an additional and duplicate request. To avoid this issue, a new custom
LinkHandler
can be used to create aChannelTabExtractor
instance with the data already fetched in it:ReadyChannelTabListLinkHandler
.Alternatively, a specific tab can be fetched using the corresponding
ChannelExtractor
methods taking a specific channel tab name (which should be one of theChannelTabs
constants) and returning channel tab extractors. In this case,ReadyChannelTabListLinkHandler
s cannot be used, so duplicate requests could happen with this process; passing an invalid or unsupported tab name may return an exception indicating that the given tab name is invalid or unsupported.Services' channel tab extractors, which so have
ChannelTabExtractor
as a base abstract class, areListExtractor
implementations allowing anyInfoItem
type. This results in the ability to provide streams, playlists and channels in channel tabs, and would allow to implement a posts item type in the future to support contents of YouTube's community tab for instance.Channel tabs could be also fetched with their
Info
equivalent,ChannelTabInfo
.Note: Contrary to the original PR, channel tabs have their own package,
tabs
, a package of thechannel
one, in whichChannelTabExtractor
,ChannelTabInfo
andChannelTabs
classes are located. TheReadyChannelTabListLinkHandler
class, which may need to be renamed, is still in thelinkhandler
package.The following channel tabs are supported with this PR:
What is done with #951 but has been not highlighted is the support of channel tags: a list of strings which can be get using the
getTags
method ofChannelInfo
andChannelExtractor
. Only YouTube seem to provide this information, so support has been added for this service only, other ones return an empty list, the default behavior.I decided to do a little bit more than copying and improving changes of the original PR in this one: I also added support of detecting age-restricted YouTube channels and getting
StreamInfoItem
s of them using YouTube's system playlists. These playlists may contain geo-restricted (and so unavailable) videos. This only works when getting tabs from a channel, due to the usage of aReadyChannelTabListLinkHandler
and aChannelTabExtractor
implementation wrapping aPlaylistExtractor
in this case. This support should be added when getting directly tabs, by moving the corresponding code as a fallback for stream items to YouTube's standardChannelTabExtractor
.So, for these channels, the Videos, Shorts and Live tabs will be always returned, even if a tab or multiple tabs do or does not exist (it/they would return no items in this case, even if system playlists don't exist when a type of a content has been not created by a channel).
Note that support of the Shorts tab using system playlists is currently broken, as YouTube return a short UI for playlists with only shorts in most cases; this UI is currently not supported by the extractor. See #1077 for more details.
Tests have been of course updated to reflect the changes made in this PR, and the YouTube channels mocks had to be updated.
Closes #227, closes #501 (only support for topic channels which have regular tabs has been added, so system channels such as Music or Gaming return no tabs, this is tracked by #611), closes #585.
Note for reviewers: I recommend to review the changes commit by commit, as they are huge.
Debug APK built by @Stypox: app-debug.zip